1. Field of the Invention
The present invention relates to an image processing system, image processing apparatus, aberration correction method, and computer-readable storage medium, which correct aberrations of an optical system.
2. Description of the Related Art
In recent years, as a technique for seamlessly blending physical and virtual worlds in real time, a so-called MR (Mixed Reality) technique is known. As one MR technique, a technique using a video see-through HMD (Head Mounted Display; to be hereinafter abbreviated as an HMD) is known. With this technique, an object which nearly matches an object observed from the pupil position of an HMD user is captured using, for example, a video camera, and CG (Computer Graphics) is superimposed on that captured image. Then, the HMD user can observe an MR image.
The video see-through HMD acquires digital image data of an object by capturing an image of that object using a charge coupled device such as a CCD, and displays an MR image (mixed reality image) superimposed with a CG image for the user via a display device such as a liquid crystal display.
Size and weight reductions of the HMD to be mounted on the head are in demand. As for image capture and display optical systems, a method of applying digital correction to various aberrations by signal processing is selected rather than a method of correcting them by optical approaches that generally lead to increases in size and weight, and approaches that adopt inexpensive lenses or reduce the number of lenses are adopted.
When an optical system is configured using inexpensive lenses or by suppressing the number of lenses, high image quality of an image to be displayed cannot often be maintained due to lens aberration. That is, barrel- or pin-cushion-shaped images are often obtained due to distortion aberrations of lenses. Also, red, blue, and green color bleeding appears at the boundaries of object images due to chromatic aberrations of magnification of lenses. For this reason, a technique is required to correct image quality drop of an object image due to such aberrations of lenses.
Techniques which correct distortion aberrations and chromatic aberrations of magnification of various aberrations of the optical systems by signal processing are disclosed. Such techniques are roughly classified into the following three techniques based on their principal methods, and an overview of each will be provided.
The first technique is correction processing of distortion aberrations and chromatic aberrations of magnification by means of address conversion. Address conversion is a method of moving a distorted image to an ideal image position based on the correspondence between an image forming position obtained by an ideal optical system and an actual image forming position which suffers the influence of aberrations in the optical system of an image capture system. Various techniques from that which stores the correspondence associated with converted positions as a table, and simply converts the correspondence (addresses) between the read and write addresses of a memory to that which holds high-precision coordinate data after conversion are available. In a display system as well, a display position is converted based on the correspondence between a pixel to be displayed and an actual display position. When such pixel conversion is done, distortion aberrations as distortions of an image can be corrected. When conversion is done for respective colors which define each pixel, chromatic aberrations of magnification as color misregistration can be corrected.
The second technique is correction processing of chromatic aberrations of magnification by means of resolution conversion. Using different variable magnifications depending on colors, enlargement or reduction processing is applied to a reference color, thus obtaining an image which suffers less color bleeding.
The third technique is correction processing of distortion aberrations using a polynomial and of chromatic aberrations of magnification by means of distortion aberration correction of respective colors. An approximation is made using a polynomial of higher degree including correction parameters as coefficients so as to calculate coordinates after conversion.
The aforementioned address conversion is known as a technique which has relatively high versatility, and can improve the coordinate conversion precision. However, the size of a reference table which stores the correspondence with the coordinates after conversion depends on the size of an original image and the conversion precision, and the table size also becomes extreme in recent high-resolution and high-image quality trends. For this reason, the following arrangement is generally adopted. That is, in general, the reference table stores representative points after decimation in place of information of all corresponding conversion points, and coordinate values between representative points are calculated by interpolation processing between representative points.
In case of chromatic aberration correction that corrects color misregistration, an arrangement which selects one of a plurality of colors as a reference color, and stores differences from the reference color for other colors, and an arrangement which reduces the table size using the symmetry of an optical system are known (Japanese Patent Laid-Open No. 8-205181).
If right and left parts or upper and lower parts are symmetric with respect to an image, reference values of the symmetric part can be generated from one direction side, and the table size can be halved. Furthermore, when both upper and lower and right and left parts are symmetric (in case of a rotation symmetry system), the table size can be reduced to ¼ of its original size.
Linear interpolation is popularly used as interpolation processing between representative points. If the interval between neighboring representative points is set not to be so broad, errors generated at the time of interpolation can be reduced. In general, interpolation errors can be reduced by adopting an interpolation formula of a higher degree when the interval between representative points remains the same. In display image sizes from SVGA to SXGA classes which are normally adopted in a conventional HMD, if an interval ranges from about 8 pixels to 16 pixels, an interpolation calculation which is satisfactory in terms of appearance can be made independently of interpolation algorithms to be adopted. If decimation is made to have an interval more than 16 pixels, errors at the time of interpolation increase. An arrangement which implements aberration correction with higher precision using Bicubic as bicubic interpolation as interpolation processing at the time of coordinate conversion is also known (Japanese Patent Laid-Open No. 2004-153323).
However, the aforementioned related techniques suffer the following problems.
As described above, in order to realize a required precision by the address conversion, representative points have to be set at a given interval. As described above, at the SXGA resolution (image size), the conversion precision often drops when linear interpolation is used as an interpolation algorithm and the interval is set to be more than 16 pixels. In order to assure higher precision, the aforementioned polynomial of a higher degree may be used. In this case, a degree 10 or more may often be required, and an HMD that is to operate in real time requires many multipliers in calculations, thus exponentially increasing the circuit scale.
Not only an HMD but also an increased resolution is required in an image capture system device and display system device. In the future, the reference table size will tend to increase, as well as the access frequency to a memory used to configure the table. For this reason, it is expected to become increasingly difficult to reduce the table size while maintaining or improving the conversion precision.
In consideration of a balance between the two requirements, that is, the circuit scale and conversion precision, a method that adopts Bicubic as a cubic formula may be used, as described in Japanese Patent Laid-Open No. 2004-153323. However, in this case, if optical lenses or prisms have uniform media and refractive indices, image forming position deviations due to aberrations tend to change very smoothly irrespective of their shapes, the number of times of image formation, and the number of lenses. As is known, Bicubic can implement high-image quality processing in an image interpolation calculation. This is because Bicubic requires a larger number of reference points than linear interpolation (bilinear) and has an effect of enhancing the sharpness of an image.
In interpolation processing at the time of coordinate conversion, it is confirmed that coordinate conversion errors increase compared to linear interpolation since an interpolation curve which produces that arbitrary sharpening effect is adopted. This means that the bicubic interpolation is not suited to interpolating image forming position deviations, which change smoothly.
As can be seen from the above description, an interpolation processing method is required that can broaden a representative point interval, that is, that requires only a practical circuit arrangement and table size (memory size) while ideally reducing conversion errors to be as small as those of a polynomial of higher degree.